Economic evaluation of a community-based intervention : measuring the value to society

Community-based participatory interventions such as those promoting health-related behaviour change present many challenges to the measurement of benefits in economic evaluation. Such interventions can influence social as well as physical well-being, and entail a multitude of outcomes both directly for intervention participants as well as indirectly for other members of the community. Furthermore, they may not achieve immediate changes in health status but their interactive and participatory nature could mean that non-health benefits are significant. The aim of this thesis is to measure the benefits of a community-based participatory intervention in rural Nepal and draw lessons more broadly for how such interventions can be valued within economic evaluation. A contingent valuation survey of a women's group intervention designed to improve maternal and newborn health was carried out in rural Nepal. Members of eleven women's groups were interviewed along with a sample of female non-members and males from the same communities. Monetary and non-monetary measures were used to elicit preferences and respondents were asked which aspects ofthe intervention they were willing to pay for: health outcomes, non-health outcomes or both. Focus group discussions were used to both inform the design of a locally relevant survey tool, to maximise content validity, and as a means of supplementing survey data with group-based discourse. Construct validity was assessed by testing for the association between willingness-to-pay and indicators of demographic and socio-economic status. Willingness-to-pay values were aggregated and combined with costs as a cost-benefit analysis. These results were compared to those of a cost-effectiveness analysis. A sensitivity analysis was conducted to assess the impact of different assumptions on total aggregate willingness-to-pay and the net benefit of the intervention. The response rate was high in all stakeholder groups indicating that the survey was well understood and acceptable to respondents. The study found that non-health benefits were valued by over 80% of respondents and thus their omission would lead to the undervaluation of such programmes. There was no significant difference between the willingness-to-pay of women's group members and non-members, suggesting that the programme generates positive externalities. Focus groups helped to improve the content validity of the survey and to achieve high response rates by enabling questions in the survey to be framed in a manner more relevant to the community. They also gave insight into the valuation context, helped to interpret values derived from the survey and highlighted the importance of trust in the payment vehicle. This thesis also shed light on some of the empirical challenges that are faced when attempting to extrapolate sample values to a larger population and deciding whose values to include

Validity of parental recalls to estimate vaccination coverage: evidence from Tanzania.

BACKGROUND: The estimates of vaccination coverage are measured from administrative data and from population based survey. While both card-based and recall data are collected through population survey, and the recall is when the card is missing, the preferred estimates remain of the card-based due to limited validity of parental recalls. As there is a concern of missing cards in poor settings, the evidence on validity of parental recalls is limited and varied across vaccine types, and therefore timely and needed. We validated the recalls against card-based data based on population survey in Tanzania. METHODS: We used a cross-sectional survey of about 3000 households with women who delivered in the last 12 months prior to the interview in 2012 from three regions in Tanzania. Data on the vaccination status on four vaccine types were collected using two data sources, card and recall-based. We compared the level of agreement and identified the recall bias between the two data sources. We further computed the sensitivity and specificity of parental recalls, and used a multivariate logit model to identify the determinants of parental recall bias. RESULTS: Most parents (85.4%) were able to present the vaccination cards during the survey, and these were used for analysis. Although the coverage levels were generally similar across data sources, the recall-based data slightly overestimated the coverage estimates. The level of agreement between the two data sources was high above 94%, with minimal recall bias of less than 6%. The recall bias due to over-reporting were slightly higher than that due to under-reporting. The sensitivity of parental recalls was generally high for all vaccine types, while the specificity was generally low across vaccine types except for measles. The minimal recall bias for DPT and measles were associated with the mother's age, education level, health insurance status, region location and child age. CONCLUSION: Parental recalls when compared to card-based data are hugely accurate with minimal recall bias in Tanzania. Our findings support the use of parental recall collected through surveys to identify the child vaccination status in the absence of vaccination cards. The use of recall data alongside card-based estimates also ensures more representative coverage estimates

Improving quality of care through payment for performance: examining effects on the availability and stock-out of essential medical commodities in Tanzania.

OBJECTIVE: To evaluate the effects of payment for performance (P4P) on the availability and stock-out rate of reproductive, maternal, newborn and child health (RMNCH) medical commodities in Tanzania and assess the distributional effects. METHODS: The availability of RMNCH commodities (medicines, supplies and equipment) on the day of the survey, and stock-outs for at least one day in the 90 days prior to the survey, was measured in 75 intervention and 75 comparison facilities in January 2012 and 13 months later. Composite scores for each subgroup of commodities were generated. A difference-in-differences linear regression was used to estimate the effect of P4P on outcomes and differential effects by facility location, level of care, ownership and socio-economic status of the catchment population. RESULTS: We estimated a significant increase in the availability of medicines by 8.4 percentage points (P = 0.002) and an 8.3 percentage point increase (P = 0.050) in the availability of medical supplies. P4P had no effect on the availability of functioning equipment. Most items with a significant increase in availability also showed a significant reduction in stock-outs. Effects were generally equally distributed across facilities, with effects on stock-outs of many medicines being pro-poor, and greater effects in facilities in rural compared to urban districts. CONCLUSION: P4P can improve the availability of medicines and medical supplies, especially in poor, rural areas, when these commodities are incentivised at both facility and district levels, making services more acceptable, effective and affordable, enhancing progress towards universal health coverage

Household costs of healthcare during pregnancy, delivery, and the postpartum period : a case study from Matlab, Bangladesh

A household survey was undertaken in Matlab, a rural area of Bangladesh, to estimate the costs incurred during pregnancy, delivery, and the postpartum period for women delivering at home and in a health facility. Those interviewed included 121 women who delivered at home, 120 who delivered in an ICDDR,B basic obstetric care (BEOC) facility, 27 who delivered in a public comprehensive obstetric care (CEOC) hospital, and 58 who delivered in private hospitals. There was no significant difference in total costs incurred by those delivering at home and those delivering in a BEOC facility. Costs for those delivering in CEOC facilities were over nine times greater than for those delivering in BEOC facilities. Costs of care during delivery were predominant. Antenatal and postnatal care added between 7% and 30% to the total cost. Services were more equitable at home and in a BEOC facility compared to services provided at CEOC facilities. The study highlights the regressive nature of the financing of CEOC services and the need for a financing strategy that covers both the costs of referral and BEOC care for those in need.This research was funded under the Cooperative Agreement No. 388-A-00-97-00032-00 with the United States Agency for International Development (USAID) (ICDDR,B Grant No. GR-00089). ICDDR,B acknowledges with gratitude the commitment of USAID to the Centre’s research efforts. Carine Ronsmans and Jo Borghi are funded by the Department for International Development, UK

Understanding causal pathways within health systems policy evaluation through mediation analysis:An application to payment for performance (P4P) in Tanzania

BACKGROUND: The evaluation of payment for performance (P4P) programmes has focused mainly on understanding contributions to health service coverage, without unpacking causal mechanisms. The overall aim of the paper is to test the causal pathways through which P4P schemes may (or may not) influence maternal care outcomes. METHODS: We used data from an evaluation of a P4P programme in Tanzania. Data were collected from a sample of 3000 women who delivered in the 12 months prior to interview and 200 health workers at 150 health facilities from seven intervention and four comparison districts in Tanzania in January 2012 and in February 2013. We applied causal mediation analysis using a linear structural equation model to identify direct and indirect effects of P4P on institutional delivery rates and on the uptake of two doses of an antimalarial drug during pregnancy. We first ran a series of linear difference-in-difference regression models to test the effect of P4P on potential mediators, which we then included in a linear difference-in-difference model evaluating the impact of P4P on the outcome. We tested the robustness of our results to unmeasured confounding using semi-parametric methods. RESULTS: P4P reduced the probability of women paying for delivery care (-4.5 percentage points) which mediates the total effect of P4P on institutional deliveries (by 48%) and on deliveries in a public health facility (by 78%). P4P reduced the stock-out rate for some essential drugs, specifically oxytocin (-36 percentage points), which mediated the total effect of P4P on institutional deliveries (by 22%) and deliveries in a public health facility (by 30%). P4P increased kindness at delivery (5 percentage points), which mediated the effect of P4P on institutional deliveries (by 48%) and on deliveries in a public health facility (by 49%). P4P increased the likelihood of supervision visits taking place within the last 90 days (18 percentage points), which mediated 15% of the total P4P effect on the uptake of two antimalarial doses during antenatal care (IPT2). Kindness during deliveries and the probability of paying out of pocket for delivery care were the mediators most robust to unmeasured confounding. CONCLUSIONS: The effect of P4P on institutional deliveries is mediated by financing and human resources factors, while uptake of antimalarials in pregnancy is mediated by governance factors. Further research is required to explore additional and more complex causal pathways

How to and how not to develop a theory of change to evaluate a complex intervention: reflections on an experience in the Democratic Republic of Congo.

Theories of change (ToCs) describe how interventions can bring about long-term outcomes through a logical sequence of intermediate outcomes and have been used to design and measure the impact of public health programmes in several countries. In recognition of their capacity to provide a framework for monitoring and evaluation, they are being increasingly employed in the development sector. The construction of a ToC typically occurs through a consultative process, requiring stakeholders to reflect on how their programmes can bring about change. ToCs help make explicit any underlying assumptions, acknowledge the role of context and provide evidence to justify the chain of causal pathways. However, while much literature exists on how to develop a ToC with respect to interventions in theory, there is comparatively little reflection on applying it in practice to complex interventions in the health sector. This paper describes the initial process of developing a ToC to inform the design of an evaluation of a complex intervention aiming to improve government payments to health workers in the Democratic Republic of Congo. Lessons learnt include: the need for the ToC to understand how the intervention produces effects on the wider system and having broad stakeholder engagement at the outset to maximise chances of the intervention's success and ensure ownership. Power relationships between stakeholders may also affect the ToC discourse but can be minimised by having an independent facilitator. We hope these insights are of use to other global public health practitioners using this approach to evaluate complex interventions